Repository navigation
feat(testing): add DeepSeek live qualification, Steward v2.4.0 eval cases, and pytest speedups - #9201
Conversation
…ases, and pytest speedups Signed-off-by: LoopX Agent <[email protected]>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · Gemini · Google · self_reported
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
在精确 head 624a2a2a6579e7f149ea02b7965ab257406b9bc7 上完整复核了 PR #9201 全部 25 个改动文件,并独立运行了核心单测、子进程回退等价性验证、正交成对覆盖数学校验、静态检查与预合并门禁:DeepSeek 实网测试入口与双全局根修复严谨,5 处测试加速全部保持原始语义且未触碰共享提示词规则与 v2.4.0.md。
动机 / Motivation
发版维护者与开发者在执行 DeepSeek 实网行为核查、Steward Golden Query 评估以及本地全量 pytest 回归测试时,会受到工具覆盖缺失与测试冷启动耗时过长的影响。改动前,行为资格套件仅支持豆包 Ark 端点,host_prompt_behavior 在同时存在双全局根目录时会触发探测异常且无法剥离 Markdown JSON 代码块,Steward 评估仅覆盖 28 条用例且默认 2 并发,本地测例则因每次夹具冷启 Node、子进程调用 CLI 及过宽的笛卡尔乘积与轮询次数耗时较久;改动后新增严格的 DeepSeek 端点与模型白名单校验、双根安全传参、30 条公开评测与默认 6 并发,并通过可自动重启的持久化 Node Worker、进程内 CLI 快速路径、隔离编译缓存及正交成对覆盖显著缩短测试耗时且未弱化断言。实测表明,共享提示词与控制面核心文件(loopx/control_plane/quota/、loopx/chat.py、loopx/heartbeat_prompt.py)及 docs/product/releases/v2.4.0.md 均为 0 行改动;DeepSeek 非规范 URL、缺失密钥、脏仓及未传 --release-live 均严格快速失败;Steward 30 条用例与 --workers 1..12 范围校验全部通过;持久化 Node Worker 自动重拉、in-process 与 LOOPX_LEASE_SCOPE 外部子进程双路径等价性、6 行成对矩阵 24/24 两两交互覆盖以及 12/24 轮询重启边界均已验证通过。本 PR 不修改核心控制面运行时语义与共享提示词契约(rules.py、task_body.py、chat.py、isolate_sqlite_runtime),不改动 docs/product/releases/v2.4.0.md,也不要求在无密钥的常规 CI 中调用外部付费模型接口。
改动思路 / Approach
DeepSeek 适配层通过参数化复用了现有直接对话/工具调用基类、传输异常分类及 5 套行为状态机(未产生平行控制面决策或私有枚举),同时所有 pytest 加速均严格限定在 tests/ 与 benchmark/tests/ 夹具边界内,核心控制面代码(quota/、chat.py、heartbeat_prompt.py)与 v2.4.0 发布说明保持零修改。当前 PR 边界严格限定在测试评估套件、实网质检脚本、测试夹具加速及对应开发/协议文档,不修改共享提示词规则、核心调度状态机或稳定版发布说明。
flowchart TD
A["DeepSeek / Eval 入口 (624a2a2a65)"] --> B{"--live / --release-live & 干净仓 & Key?"}
B -- "缺参或脏仓" --> B1["FAIL-CLOSED: skip(0 调用) 或 RuntimeError"]
B -- "非规范 BASE_URL" --> B2["noncanonical_endpoint 拒绝请求"]
B -- "通过" --> C["复用 _DirectChat* / DeepSeekExecToolClient / 5 状态机"]
D["Pytest 加速路径"] --> E["_CanonicalAuthorityWorker 按 (pid,pkg,sqlite) 复用与 1 次自动重拉"]
D --> F["test_canonical_lease_scopes 默认 cli_main,有 cwd/env 时回退子进程"]
D --> G["6 行成对矩阵 (24/24 两两覆盖) & 12/24 轮询 (达 3600s)"]
基于精确 head 624a2a2a6579e7f149ea02b7965ab257406b9bc7 源码与本地检查整理,仅作控制流示意。
| 核查面 | 基线 (0a0c2a7c8b) |
当前 head (624a2a2a65) |
独立验证结论 |
|---|---|---|---|
| 端点/密钥/脏仓安全门 | 仅支持 Ark,无 DeepSeek 端点规范 | resolve_deepseek_chat_completions_endpoint 仅放行官方根路径//v1 |
PASS (非规范 URL/缺密钥/脏仓均拦截) |
| 双全局根与围栏 JSON | probe_messages 未传 registry_path |
显式传入合成 Path('registry.json') + _strip_json_markdown_fence |
PASS (test_host_prompt_behavior.py 通过) |
| Steward 评估集与并发 | 28 条用例,固定 max_workers=2 |
30 条用例,--workers 支持 1..12(默认 6) |
PASS (test_chat_intake_evaluation.py 通过) |
| 持久化 Node Worker | 每次初始化/推进均冷启 Node | 进程级带锁 _CanonicalAuthorityWorker(TS 依赖 0 process.env) |
PASS (91 passed in 4.35s + 自动重拉) |
| CLI / 正交 / 轮询加速 | 始终子进程 / 16 组笛卡尔 / (100,200) 轮 |
默认 cli_main + LOOPX_LEASE_SCOPE_* 回退;6 行 24/24 覆盖;(12,24) |
PASS (子进程对比 2/2、成对 6/6、轮询 2/2) |
具体改动 / Changes
本次基于公共分叉基线 0a0c2a7c8bb38bad503bb3eba47cb4a3f0974b4d 共改动 25 个文件(+1593 / -205 行):实网评估与测试适配器(+757 / -127)、测试加速与新增单测(+772 / -72)及 4 份文档更新(+64 / -6);确认 docs/product/releases/v2.4.0.md、loopx/control_plane/quota/、loopx/chat.py、loopx/heartbeat_prompt.py 均为 0 行改动。
对照协议规范 docs/reference/protocols/model-behavior-qualification-v0.md(精确修订 624a2a2a6579e7f149ea02b7965ab257406b9bc7):
DS-QUAL-1(Direct DeepSeek 对话与 5 类工具行为角色严格限定官方端点、禁用 thinking、模型白名单并复用共享状态机):implemented(通过test_actual_default_model_behavior_live.py及脏仓/缺密钥反事实验证)。DS-QUAL-2(Host prompt 发布探针支持--provider deepseek、默认关闭、剥离 JSON 围栏且免疫双全局根冲突):implemented(通过test_host_prompt_behavior.py与默认跳过反事实验证)。STEWARD-EVAL-1(Steward 公开集扩展至 30 条含 PR 状态与监控只读区分用例,并支持--workers 1..12):implemented(通过test_chat_intake_evaluation.py验证)。PYTEST-SPEED-1(测试加速保持 worker/路径隔离、子进程回退、全量两两交互覆盖与 3600s 退避重启回归不变):implemented(通过 7 套单测、子进程等价跑测、24/24成对校验及12/24轮询验证)。
关键代码讲解
loopx/control_plane/testing/doubao_model_behavior_actor.py—resolve_deepseek_chat_completions_endpoint(L89-101)与_direct_chat_completions_transport(L104-169):规范化DEEPSEEK_BASE_URL仅接受官方根 URL 与/v1,其它地址以noncanonical_endpoint阻断;抽取公共 HTTPS 传输路径禁用重定向并统一脱敏401鉴权报错,避免暴露密钥。loopx/control_plane/testing/doubao_model_behavior_actor.py—_strip_json_markdown_fence(L456-472):仅剥离外层空标签或json围栏代码块并同时供_provider_decision与host_prompt_behavior.run_probe复用,对内部非法 JSON 仍保留严格解码拦截。loopx/control_plane/testing/model_tool_behavior.py—DeepSeekExecToolClient(L343-389)及 5 处from_deepseek_environment:通过可选endpoint/allowed_models/provider_prefix参数复用DoubaoExecToolClient,令selected_todo、replan_semantic_action、scoped_gate_successor、capability_monitor_repair与terminal_settlement共用同一状态机。loopx/control_plane/testing/host_prompt_behavior.py—probe_messages(L59-86):向build_heartbeat_prompt显式传递合成registry_path=Path("registry.json"),阻断在同时存在~/.loopx与~/.codex/loopx全局目录时触发隐式多根歧义报错。tests/control_plane/canonical_authority_fixture.py—_CanonicalAuthorityWorker(L31-197):按(pid, package, sqlite_enabled)维护带线程锁与45s选择器超时的持久化 Node 进程,注册atexit回收并支持异常退出后 1 次自动重拉;已核查其加载的 4 个 TypeScript 依赖模块无任何process.env/os.tmpdir读取。
对主干的风险 / Risks
重点风险核查结论如下:
- 夹具持久化状态污染风险:已核查
local_authority_provider.ts、file_authority_store.ts、legacy_writer_fence.ts、authority_store_codec.ts均不读取环境变量或系统临时目录,路径全部来自单次 JSON 请求显式传入的req.root与req.goal;tests/conftest.py仅在每个 pytest worker 独立临时目录创建0o700编译缓存并服从关闭标志;test_canonical_lease_scopes.py在cwd或LOOPX_LEASE_SCOPE_*存在时继续走subprocess.run。 - 测试缩减覆盖风险:经脚本逐 pair 核算,
benchmark/tests/test_task_entry.py的 6 行参数矩阵覆盖了全部24/24个两两因素组合;对照scripts/external_scheduler_worker.py的select_interval,12/24轮询仍越过重启门槛与最大3600s退避台阶。 - 其他评审维度:
scope_fit(verified)所有新增脚本、参数与夹具均有活跃调用点;default_off_isolation/authority_semantics/semantic_alignment为not_applicable(无新增未注册控制面枚举,外部请求须显式--live/--release-live+ 密钥触发);Steward 与测试默认值变化均已同步披露文档。
独立本地验证结果:
- Part A + 权威夹具单测(
test_actual_default_model_behavior_live.py、test_host_prompt_behavior.py、test_chat_intake_evaluation.py、test_canonical_authority_fixture.py):91 passed in 4.35s(0跳过)(PASS) - 7 文件合并套件与重点回归:
352 passed, 54 skipped(跳过均位于test_task_entry.py不相关平台/live 项);6 个非跳过文件收集核对305条全过;LOOPX_LEASE_SCOPE_PARITY=1子进程对照2 passed in 2.54s;test_execute_wake6 行矩阵6 passed in 155.81s;12/24依赖等待测试2 passed in 12.61s(PASS) - 静态与边界门禁:
ruff(20 文件)、mypy(17 文件)、check_rfc_bilingual.py、release-readiness-doc-smoke.py、semantic-vocabulary-drift-smoke.py(29/29 词表)、loopx check(8 根路径 / 1562 文件0错,15 条既存非字面量告警)、共享控制面/说明零改动核对:全部通过 (PASS) loopx canary premerge --from-git-diff --git-diff-base origin/main --timeout-seconds 300:选中19/19项检查全部通过(failures=0;manual_hold_count=1来自修改benchmark/tests/test_task_entry.py触发的benchmark_sensitive提示)(PASS)- 脏仓拒绝(
rc=1)、缺DEEPSEEK_API_KEY(rc=1)、未传--release-live默认跳过(status='skipped',provider_call_count=0)、24/24成对矩阵反事实:全部通过 (PASS) - 实网付费 DeepSeek API 请求:评审期间未主动消耗外部额度重跑实网,作者在 PR 描述已如实记录通过数及
notifying_wait/ Steward 现网基线差距 (UNVERIFIED);远程 GitHub CI 依 Goal 配置未等待 (UNVERIFIED)。
我的整体评价 / Verdict
基于独立校验,PR #9201 在严格保持共享控制面与提示词规则零改动、且隔离测试安全的前提下,完整补齐了 DeepSeek 行为验证与 Steward v2.4.0 公开评估能力,并显著加速了慢速 pytest 套件。problem_context.outcome_impact 评估:long_horizon 为 improved(进程级带锁持久化 Node Worker、严格端点与凭证门禁、跨重启 3600s 退避测试均保持稳健);user_experience 为 improved(维护者获得统一的 --provider deepseek 门禁与双全局根修复且保留显式串行/子进程回退开关);visual_review 为 not_applicable:已复核全部 25 个改动文件,均属于评估脚本、测试框架夹具、行为模拟模块与协议/测试文档,无前端视图或交互式 UI 页面变动;未验证范围(unverified_scope):none。
observable_semantics 判定为 intentional_change_validated(chat-intake --workers 6 / 30 用例、6 行 24/24 成对矩阵与 (12,24) 轮询调整均有测试与文档披露并保留显式回退路径),scope_coverage 判定为 verified,code_volume 为 necessary(兼容性 retain),change_proportionality 为 proportionate 且机制评估 retain,default_off_isolation / authority_semantics / semantic_alignment 均为 not_applicable(无新控制面状态词表,29/29 词表检查通过)。后续可非阻断地抽离 5 个工具行为类中重复的 from_deepseek_environment 模板;注意合并时 benchmark/tests/test_task_entry.py 触发 benchmark_sensitive 提示由授权维护者核实。精确 head 624a2a2a6579e7f149ea02b7965ab257406b9bc7 质量达标,建议批准。
English verdict: APPROVE on exact head 624a2a2: shared prompt/runtime contracts and v2.4.0 release notes have 0 diff lines, DeepSeek endpoints/actors/dual-root/fence fixes and test speedups pass all local suites and counterfactuals (91 Part A+fixture tests PASS in 4.35s, 305 targeted non-skipped tests PASS, LOOPX_LEASE_SCOPE_PARITY subprocess parity PASS, 24/24 pairwise coverage PASS, semantic-vocabulary-drift-smoke PASS, canary premerge 19/19 PASS with manual_hold_count=1 for benchmark_sensitive test matrix reduction); live paid DeepSeek API calls remain un-re-run in review.
Summary
Adds opt-in live DeepSeek model-behavior & Steward evaluation support plus vision-aligned pytest speedups without mutating shared production prompt/runtime contracts (
rules.py,task_body.py,chat.py,isolate_sqlite_runtime) ordocs/product/releases/v2.4.0.md.Part A — DeepSeek Live Behavior Qualification & Steward Evaluation Coverage
loopx/control_plane/testing/doubao_model_behavior_actor.py: Keeps allDOUBAO_*,_direct_ark_transport, andDoubao*Actorclasses intact while adding DeepSeek constants (DEEPSEEK_FLASH_MODEL="deepseek-flash",DEEPSEEK_V4_PRO_MODEL="deepseek-v4-pro",ALLOWED_DEEPSEEK_BEHAVIOR_MODELS),resolve_deepseek_chat_completions_endpoint(canonicalhttps://api.deepseek.com/chat/completions, failing closed withnoncanonical_endpointon non-canonical URLs),_direct_deepseek_transport(_NoRedirectHandler),_strip_json_markdown_fence,DeepSeekModelBehaviorActor, andDeepSeekOnboardingModelBehaviorActor(actor_ref="deepseek:<model>", defaultdeepseek-flash). Zero unownedLiteral/Enumtypes introduced.loopx/control_plane/testing/model_tool_behavior.py+ 5 tool-behavior actors: ParameterizesDoubaoExecToolClientwith optionalendpoint,allowed_models, andprovider_prefix, addsDeepSeekExecToolClient, and adds optionalclient=None+from_deepseek_environment(...)to all 5 tool-behavior actors (selected_todo,replan_semantic_action,scoped_gate_successor,capability_monitor_repair,terminal_settlement) reusing existing state machines and semantic oracles.scripts/qualify-deepseek-model-behavior-live.py,scripts/qualify-host-prompt-release.py,host_prompt_behavior.py: Addsscripts/qualify-deepseek-model-behavior-live.py(--max-workers 6default), extendsscripts/qualify-host-prompt-release.pywith--provider {ark,deepseek}and--model, passes syntheticregistry_path=Path("registry.json")inhost_prompt_behavior.py::probe_messagesto prevent dual global-root lookup crashes on multi-host machines, and strips optional JSON markdown fences inrun_probe.examples/evaluations/chat-intake.py,chat-intake.public.json,docs/product/use-cases/steward/golden-queries.md: Adds bounded--workers(1..12, default6) and 2 public-safe Steward cases (unconfirmed-pr-merge-status-only,watch-only-monitor-not-completed; 28 → 30 cases), plus DeepSeek unit coverage intests/control_plane/test_actual_default_model_behavior_live.py,tests/control_plane/test_host_prompt_behavior.py, andtests/test_chat_intake_evaluation.py.docs/reference/protocols/model-behavior-qualification-v0.md,docs/development/testing-and-quality.md,docs/product/release-readiness.md,docs/product/use-cases/steward/golden-queries.md(docs/product/releases/v2.4.0.mdintentionally left untouched).Part B — Vision-Aligned Pytest Speedups (Zero Invariant Weakening)
tests/conftest.py::_isolated_worker_effect_runtime: Sets scoped0o700NODE_COMPILE_CACHEinside each worker's isolated temporary runtime unlessNODE_DISABLE_COMPILE_CACHE=1orNODE_V8_COVERAGEis set.tests/control_plane/canonical_authority_fixture.py: Replaces repeated per-call Node cold-starts ininitialize_canonical_authority/advance_canonical_fixture_authoritywith a thread-locked persistentPopenNode worker per(pid, package_root, sqlite_enabled)withatexitcleanup and 1x auto-respawn while keepingisolate_sqlite_runtimeunchanged (test_canonical_authority_fixture.py: 7 + 2 new tests = 9 passed).tests/control_plane/test_canonical_lease_scopes.py: Routes clean default CLI calls in-process vialoopx.cli.main(redirect_stdout), retainingsubprocess.runwhenevercwdorLOOPX_LEASE_SCOPE_*is overridden.benchmark/tests/test_task_entry.py::test_execute_wake_keeps_host_clocks_out_of_fresh_and_resumed_input: Replaces 16x full Cartesian product with a 6-row pairwise covering matrix preserving the[900, 500]clock loop.tests/control_plane/test_dependency_vision_wait_cli.py: Reduces wait/restart polling iterations(100, 200)→(12, 24)(len(sleeps) == 24 and sleeps[-1] == 3600), still crossing 3-poll limit + saturated 3600s ticks across restart.Verification
Exact Head (
624a2a2a6579e7f149ea02b7965ab257406b9bc7onorigin/main0a0c2a7c8b)ruff checkon all 20 touched Python files: passed (0errors)mypy:Success: no issues found in 17 source filesscripts/check_rfc_bilingual.py --base origin/main --head HEAD: passedexamples/release/release-readiness-doc-smoke.py: passedloopx checkacross all 12 touched directory/file roots (795files):ok: True,errors=07files):132 passed in 5.38spersonal-loopx-maintenance):cqr_9216b09eb905ec64d47f(status: valid,decision: pass,blocking: false, scope9216b09eb905ec64d47fbf39d601e968b5071ecaf82ec9059d8d43a04d76d882)loopx canary premerge --from-git-diff --git-diff-base origin/main --goal-id personal-loopx-maintenance --timeout-seconds 300:4/4direct checks passed,1/1py_compilepassed,19/19selected checks passed (10/10catalog canaries incl.semantic-vocabulary-drift-smoke.py+control-plane-maintainability-ratchet-smoke.py,8/8risk profile smokes,1/1public boundary),0failures,quality_receipt_failure_count: 0,manual_hold_count: 1(benchmark_sensitiveonbenchmark/tests/test_task_entry.py)Additional Pre-Rebase / Exact-Code Suite & Timing Evidence (same PR diff)
14files on3688ec61cb):384 passed(132in6.07s+121in194.20s+131in234.20s,0failed,0skipped)0skipped/relaxed assertions):tests/control_plane/test_canonical_authority_fixture.py:2.80s(7 passed) →1.73s(7 passed, quiet) /5.69s(9 passed incl. 2 new worker tests on3688ec61cb)tests/control_plane/test_dependency_vision_wait_cli.py:110.76s(20 passed) →40.44s(quiet) /69.61s(20 passed under concurrent release lanes on3688ec61cb)tests/control_plane/test_canonical_lease_scopes.py:302.79s(194 passed) →113.46s(quiet) /134.40s(194 passed under load on3688ec61cb)benchmark/tests/test_task_entry.py -k test_execute_wake_keeps_host_clocks_out_of_fresh_and_resumed_input:408.23s(16 passed) →152.69s(quiet) /161.27s(6 passed under load on3688ec61cb)824.58s→308.32s(-62.6%, quiet) /370.97s(-55.0%, under concurrent load)Live DeepSeek Qualification & Steward Evals (Honest Unfiltered Baseline)
qualify-deepseek-model-behavior-live.py --max-workers 6):deepseek-flash:22/22scenarios &6/6contrasts passed (qualification_passed=True) in97calls (3688ec61cb),100calls (379fa74955), and101calls (9911d1162c); on624a2a2a65scored21/22scenarios &6/6contrasts (128calls; repeat 1 passed in11calls, repeat 2 ofturn_required_vision_replanhittool_call_budget_exhaustedat40/40calls withshell_nonzero: 2).deepseek-v4-pro(3688ec61cb):17/22scenarios &6/6contrasts passed (133calls; failed:onboarding_agent_identity_gate,turn_terminal_settlement,turn_planning_horizon_strategic_context,turn_required_vision_replan,turn_scoped_gate_successor_replan).qualify-host-prompt-release.py --release-live --provider deepseek --repeats 2:deepseek-flash:17/20passed (4/4quiet_work,quiet_wait,vision_replan,wait_fallback_work; failed onlynotifying_wait:thinx2,briefattempt 2).deepseek-v4-pro:16/20passed (4/4on those 4 cases; failednotifying_waitonthinx2,briefx2).rules.py/task_body.pyuntouched per scope): both models interpretedquota_skip+agent_channel.delivery_allowed=false+"Output only under user_channel.notify=NOTIFY; otherwise stay quiet"as needing no user output even whenoperator_questionandNOTIFYare present.examples/evaluations/chat-intake.py --live --provider operator-api --workers 6 --repeats 2):deepseek-flash:chat-intake 48/60(unconfirmed-pr-merge-status-only2/2,watch-only-monitor-not-completed1/2),chat-routing 5/18,chat-followthrough 7/14,chat-material 6/14,chat-purpose 1/8.deepseek-v4-pro:chat-intake 47/60(unconfirmed-pr-merge-status-only2/2),chat-routing 6/18,chat-followthrough 5/14,chat-material 10/14,chat-purpose 4/8.